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ABSTRACT 

In 1992, the Leon County School District in 
Tallahassee (Florida) began a review process to select a new 
norm-referenced test. The selection process was efficient and 
functional, and may be of interest to other school districts. 
Consideration was given to a variety of sources through the following 
process steps: (1) establish the norm-referenced test steering 
committee; (2) establish the process for test review and selection, 
objectives for the testing, and criteria for test selection; (3) 
conduct a technical and preliminary test review; (4) conduct a 
content review for critical curriculum match; (5) review 
presentations by publishers; (6) provide service documentation from 
users and publishers' negotiations; and (7) produce a final 
recommendation, pilot and phase-in plan, and teacher review of final 
selection. Evaluations by Steering Committee members confirm that the 
selection process was effective, fair, and efficient. Five appendixes 
contain summaries of objectives, criteria, and the process; a test 
review rating chart; and the curriculum/test match worksheet. 
(SLD) 
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In 1992 , the Leon County School District in Tallahassee, Florida, 
began a review process to select a new norm-referenced test. The 
process proved efficient and functional and may be of interest to 
other public school districts anticipating test adoptions. 

In designing a review model and recognizing that our ultimate 
objective was the selection of a test which would best meet the 
needs of Leon County Schools, consideration was given to the 
inclusion of input and direction from a variety of sources. This 
was accomplished through: 

Step #1, Establish the Norm-Referenced Test Steering Committee . 

Central to the entire process was the establishment of a Steering 
Committee. The committee was composed of representatives of key 
groups across district divisions and levels — school and 
administrative. This group served as a guiding force throughout 
the process and met at critical points during the review to discuss 
issues, problems, concerns, direction, status, and recommendations. 

Once the committee was established, personnel of the Department of 
Student Assessment and Database Reporting developed a model for 
review and selection of a new test, including selection criteria 
and our (local) objectives for norm-referenced testing. This work 
met the requirements of: 

Step f2. Establish the Process for Test Review and Selection, the 
Objectives for Norm-Referenced Testing, and the Criteria for Test 
Selection. Drafts of these documents were shared with the Steering 
Committee. Using their input, the documents were revised and 
finalized (see Appendices 1, 2, and 3). These documents shaped the 
process by providing a framework within which to work as well as 
important conceptual fundamentals around which decisions regarding 
elimination or acceptance were made. 

With the establishment of the committee, the model, the criteria, 
and the objectives, several tests in the review pool were 
eliminated (for example, those with norms which were three years 
old or older) . 
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In order to determine which tests were of high technical quality 
(item #1. on list of criteria) , we began: 

Step #3. Technical and Preliminary Test Review. An externa 1 
consultant was hired by Leon County Schools to conduct a technical 
review based on certain prescribed parameters set forth by the 
school district (i.e., based on the AERA/APA/NCME Standards for 
Educational and Psychological Testing and was to consider Leon 
County Schools' Objectives for Nationally-normed Testing) . 
Briefly, once the review was conducted, publishers were asked to 
respond to any rating other than "adequate" — those marked 
"questionable, 11 "inadequate," or left blank because insufficient 
information was provided. Those items were then re-evaluated for 
possible rating changes. This "second round" was included in the 
event that information was missing from the review sets or was 
misinterpreted by our consultant. Our intent, then, was to be as 
fair as possible and allow publishers to "defend" their tests on 
specific technical qualities set forth through our descriptive 
specifications. Additionally, our consultant participated in an 
interactive capacity throughout the remaining portion of the review 
to answer questions and provide information from a technical 
perspective. 

Following the preliminary and technical reviews, all but four tests 
(the most current from the four major vendors) were eliminated, 
only to be revisited if none of the remaining tests were able to 
meet Leon County Schools' remaining criteria. 

Although we realize that "national" norm-referenced tests purport 
to sample from a "national" curriculum, it was important to us to 
consider how well each test "fit" our own curriculum. This was 
addressed in: 

Step #4. Content Review for Critical Curriculum Match. Elementary 
and secondary teachers representing major curriculum areas 
(reading/language arts, mathematics, science, social studies, 
Chapter 1, and Exceptional Student Education) and district 
curriculum coordinators were provided test materials and district 
curriculum frameworks to review for recommendations of which test 
provided the best "fit" to our district curriculum. Review forms 
were completed and used as a basis for summary and discussion (see 
Appendices 4 and 5) . As a follow-up, and again for the sake of 
fairness in the event of a "fatal" problem, publishers were invited 
back for a question and answer session with teacher representatives 
and curriculum coordinators . Following the question and answer 
sessions, teachers 1 recommendations for best fit were finalized. 
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All areas of our criteria for selection which were not addressed in 
the curriculum match were addressed in: 

Step #5, Publishers 1 Presentations. Each of the four publishers 
provided two-hour presentations to the steering committee. 
Publishers were notified in advance of the Objectives for Testing, 
the Criteria for Test Selection, and the specific criteria upon 
which their presentations should focus (see Appendix 6) . Committee 
members were provided review forms (see Appendix 7) whirh were 
completed and used as a basis for discussion, summary, and 
recommendation following the presentations. Based on committee 
findings, three tests were carried forward for further study at 
this point. 

Prior to a final recommendation (and in order to have mor° "room" 
to negotiate with publishers) , we addressed business issues such as 
service reliability and responsiveness and cost via: 

Step #6, Service Documentation from Users; Publisher Negotiations. 

As the process began to move toward a final recommendation, data 
was collected from product users and specific cost and materials 
proposals were requested of publishers. 

These findings, in addition to all previously gathered information, 
were presented to and discussed by the Steering Committee to help 
the committee come to: 

Step #7. Final Recommendation; Pilot/Phase-In Plan; Teacher Review 
of Final Selection; implementation. Based on the Steering 
Committee's recommendation, a proposal for an adoption, including 
a pilot/phase-in plan, was presented to appropriate district 
decision makers. Following approval, a plan to provide inservice 
to teachers regarding the new test was put in place. A combination 
of methods was used including on-site visitations to individual 
schools, multiple presentations at a single site on a district-wide 
inservice day, and video airing through our school district's 
instructional television network. 

Schools have been encouraged to work closely with parents and have 
provided their own presentations to parents as well as inviting 
district personnel to make presentations to parent groups. 

The actual test implementation process is currently in progress. 

Leon County drew from a number of data sources in designing and 
implementing the review process. Most heavily utilized were 
revised review procedures from other school districts such as those 
from Hillsborough County, Orange County (both in Florida) and 
former Leon County review processes. 
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In conclusion, and based on evaluations from Steering Committee 
members (representing critical areas affected by norm-referenced 
testing) , it was felt that the process was an effective, fair, and 
efficient one. By sharing this model, it is hoped that knowledge 
of Leon County's successful implementation of this model may assist 
others anticipating test adoptions. 



Leon County Schools 

Objectives for Nationally-Normed Testing 
(Why are we testing?) 



Leon County Schools 1 nationally-normed testing should: 

1. assess student achievement of objectives which are considered 
to be representative of a "nationally common" curriculum; 

2. provide a valid and reliable basis for assessing relatively 
low to high levels of achievement based on national norms; 

3. provide a continuous and comparable longitudinal 
assessment structure to assess growth of individual students 
or groups of students from grade to grade and from level to 
level (i.e., from elementary to middle or middle to high 
school) through the use of a single achievement test; 

4. provide information to help screen (for selection), monitor, 
and evaluate various educational programs, projects, 

and curricula; 

5. provide achievement level feedback to students, parents, 
teachers, instructional program leaders, the State Department 
of Education, the media, and the community as a whole; 

6. screen individual achievement/ aptitude relative to special, 
remedial, and/or advanced instructional group placements and 
course scheduling. 
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Criteria for National ly-Normed Test Selection 

To be considered for adoption in Leon County, a national ly-normed 
test must: 

1. be of high technical quality in terms of APA/AERA/NCME 
standards for tests; 

2. be non-biased in terms of sex, or racial/ethnic references; 

3. be normed on a recent national sample of students (as defined 
by the NRT Steering Committee) ; 

4. have sufficient sub-test depth to allow a reasonable 
progression of percentiles (i.e., one raw score point 
doesn't cause the corresponding national percentile to jump 
+/- 15 to 20 points) ; 

5. be constructed in such a way as to reduce ceiling and floor 
effects which greatly inhibit a test's usefulness in dis- 
criminating upper and lower achieving students; 

6. provide sub-test (content) coverage in reading, language, 
and mathematics; 

7* be matched as closely as possible to Leon County Schools 1 
curriculum in critical areas as determined by executive 
directors; 

8. include an aptitude test which will generage anticipated 
achievement scores; 

9. generate useable score types (national percentiles, standard 
scores, grade equivalents, normal curve equivalents, local 
percentiles) and useable summary information (means, 
medians , standard deviations , distributions , quart ile 
summaries) which are derived in a technically sound and 
locally acceptable manner in report formats which are easy to 
read and interpret; 

10. allow for optional local scoring at no extra cost to the 
district other than purchasing of norms (including interpolated 
norms) ; 

11. be relatively easy to administer (within reasonable time 
constraints) and provide support materials which assist in test 
administration and interpretation (complete manuals, norms 

— including interpolated norms, technical manuals, etc,)? 

12. be of reasonable cost and be constructed of durable (reuseable) 
materials; 

13. be backed by reputable, reliable, responsive company/ publisher 
willing to provide support and assistance on request. 

Appendix 3 
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STANDARDIZED TEST REVIEW RATING CHART 
Rater 1 s Name __ 



Position 



Test Name 
Test Level 



Grades Covered 



Portions Reviewed 

Reading/ L • A • 

Mathematics 

Other 



KEY: 



0 - Poor/nonexistent/ inappropriate 

1 - Weak/ somewhat inappropriate 

2 - Adequate/appropriate 

3 - Good/very appropriate 



Test Content 
Subtests 



(A) 
Fill In If 
DOT. Nam* 



(B) 
Cognitive 
Domain 



(C) 
Item- 
Match 
W/Currtc 



(D) 
Extent 
Curr Cov 



Reada- 
bility 



Clarity 



Comments* 



Vocabulary 
Reading Comp. 



0 12 3 

0 12 3 

0 12 3 

0 12 3 



0 12 3 

0 12 3 

0 12 3 

0 12 3 



0 12 3 

0 12 3 

0 12 3 

0 12 3 



0 12 3 

0 12 3 

0 12 3 

0 12 3 



0 12 3 

0 12 3 

0 12 3 

0 12 3 



(1) Total Reading 



Spelling 

Grammar 

Caplt/Punctuatlon 



0 12 3 



0 12 3 



0 12 3 



0 12 3 



0 12 3 



0 12 3 
0 12 3 
0 12 3 



0 12 3 
0 12 3 
0 12 3 



0 12 3 
0 1 2 3 
0 12 3 



0 12 3 
0 12 3 
0 12 3 



0 12 3 
0 12 3 
0 12 3 



(2) Total Language 



Computation 
Concepts 
Problem Solving 



(3) Total Math 



0 12 3 



0 12 3 



0 12 3 



0 12 3 



0 12 3 



0 12 3 
0 12 3 
0 12 3 



0 12 3 
0 12 3 
0 12 3 



0 12 3 
0 12 3 
0 12 3 



0 12 3 
0 12 3 
0 12 3 



0 12 3 
0 12 3 
0 12 3 



0 12 3 



0 12 3 



0 12 3 



0 12 3 



0 12 3 



(4) Science 



0 12 3 



0 12 3 



0 1 2 3 0 1 2 3 



0 12 3 



(S) Social Studies 



0 12 3 



0 12 3 



0 1 2 3 0 1 2 3 



0 12 3 



Others 

(D 

(2) 

(3) 



♦Addluonai Comments: 



0 12 3 
0 12 3 
0 12 3 



0 12 3 
0 12 3 
0 12 3 



0 12 3 
0 12 3 
0 12 3 



0 12 3 
0 12 3 
0 12 3 



0 12 3 
0 12 3 
0 12 3 
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STANDARDIZED TEST REVIEW FORM 
Leon County Schools 



GENERAL DIRECTIONS: 

1. All information requested at the top of each form should be 
completed before starting the review process. 

2. Subtests; This column contains the names of subtests commonly 

found in standardized achievements tests. The subtests 

are grouped by major areas: 

Total Reading (1) 

Total Language (2) 

Total Mathematics (3) 
If other or additional subtests exist, please list each at the 
bottom in the area labeled "other". Assign the area 1, 2, or 3 
and write the title of the subtest. 



TEST CONTENT: 

Test content should be reviewed according to how well it matches Leon 
County Schools 1 curriculum. Each test should be reviewed in relation 
to corresponding grade level. 

A. Complete if Different Name: This column is to be completed only 
if the name of the subtest is different form the name given in 
the first column but the subtest meaning has not changed (i.e., 
"Vocabulary" could be listed as "Word Knowledge") . NOTE: lines 
are given below subtest names for additional subtest areas. 

B. Sampling of Cognitive Domain: This column deals with the levels 
at which items measure the concepts covered in the test. For 
example, are all items asking the student to recall facts or are 
the students asked to apply or synthesize information? (Refer to 
Bloom's Taxonomy of the Cognitive Domain .) For a good rating in 
this column, higher as well as lower levels should be 
represented. 

C. Item-Match With Curriculum: This column deals with the extent to 
which the items are consistent with the curriculum. Is the con- 
tent of the item normally taught in that grade in Leon County 
Schools? 



D. Extent of Curriculum Coverage: This column deals with whether ^ 
the items represent the range of the curriculum continuum within 
the grades the test purports to measure. This is different than 
the item-match in that every item could be included in the curri 
culum and yet not cover the breadth of the curriculum continuum. 

E. Readability of Items for Grade (s) Covered: This column deals 
with whether the items are written at a level students will 
understand. Are the words used and sentence construction too 
difficult, too easy for the majority of the students? 

Item Clarity: This column deals with whether the student 
understands what the item is asking the student to do. In 
addition, each multiple choice item should contain distractors 

which are plausible answers. i . 

1 1 A: F/sbo/wp/TESTREVI . dir 
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curriculum/Test Match Worksheet 

TEST FORM 

CURRICULUM AREA QEAPB 

Curriculum Outline Test Content Outline 

Major Minor Major Minor 

Objectives Objectives Objectives Objectives 
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Norm Referenced Testing Review Process 
Publishers 1 Presentations: Objectives & Issues 

Objective: To provide additional information to the norm- 
referenced testing steering committee which will promote their 
informed decision-making regarding the further study of or 
elimination of applicable tests as part of the review and selection 
process • 

Issues/ items to be addressed by publishers 1 in their presentations: 

According to Leon County Schools 1 criteria for nationally-normed 
test selection, a nationally-normed test/publisher must: 

1. have sufficient sub-test depth to allow a reasonable progression 
of percentiles (i.e., one raw score point doesn f t cause the 
corresponding national percentile to jump +/- 15 to 20 points) . 

2. be constructed in such a way as to reduce ceiling and floor 
effects which may inhibit a test 1 s usefulness in discriminating 
upper- and lower-achieving students. 

3. be matched as closely as possible to Leon County Schools 1 
curricula in critical areas. (Representatives of the curriculum 
review team will present their findings to the publishers, ask 
questions, and allow for defense and/or rebuttal.) 

4. be relatively easy to administer (in terms of format and 
administration procedures) within reasonable time constraints and 
provide support materials which assist in test administration and 
interpretation (complete manuals, norms — including interpolated 
norms, technical manuals, etc.). 

5. provide written commitment to the district stipulating the 
company 1 s agreement to make whatever arrangements are necessary to 
allow for local scoring with Testmate. 

6. provide written commitment to provide support and assistance 
(from publisher/consultant) in a timely and appropriate manner. 

7. provide written commitment for flexible, pro-rated payment 
schedule (possibly over three years) covering initial test 
purchase. 

Publishers are directed to focus on these points during their 
presentations, to provide backup materials which document their 
statements, and to provide rebuttal when questioned. 

Worksheets will be provided for meeting participants to use in 
rating publishers across each of the respective areas. 

a:mb/wp/nrtppobj .doc 
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STANDARDIZED TEST REVIEW FORM % . ' 

Leon County Schools 

GENERAL DIRECTIONS: 

1. All information requested at the top of each form should be 
completed before starting the review process. 

2. Subtests: This column contains the names of subtests commonly 

found in standardized achievements tests. The subtests 

are grouped by major areas: 

Total Reading (1) 

Total Language (2) 

Total Mathematics (3) 
If other or additional subtests exist, please list each at the 
bottom in the area labeled "other". Assign the area 1, 2, or 3 
and write the title of the subtest. 

RATING CRITERIA: 

A. Format : Color/Illustrations : 

Color - Is print color acceptable or distracting to the student? 
Illustrations - Are illustrations appropriate and clear? Do they 
add to the students 1 understanding or do they distract the 
student? 

B. Format: Print Size/Item Layout: 

Print size - Is the print size (for both Items and Directions) 
appropriate for the grade (s) tested? 

Item Layout - Are the items arranged in a logical, easy-to-follow 
layout? Are items spaced appropriately (not too crowded)? 

C. Directions: For Student: 

Clarity of Directions - Are the directions written so that they 
are clear in terms of what is expected? 

Sample Items - Are sample items appropriate and clearly written? 

D. Directions: For Teacher/Test Administrator: 

Clarity of Administration - Are directions for teachers/test 
administrators clearly written and easy to understand and follow? 
Information in Manuals - Does the manual adequately and usefully 
explain subtests, score interpretations, norms, etc.? 

E. Subtest Depth: 

Do tests have sufficient depth to allow a reasonable progression 
of percentiles (i.e., one raw score point doesn't cause the 
corresponding national percentile to jump +/- 15 to 20 points)? 

F. Reduced Floor and Ceiling Effects: 

Is there a sufficient number of relatively easy and difficult 
items to discriminate between the lowest and highest achieving 
students? 

G. Preparation for Testmate: 

Has the publisher provided written commitment to make whatever 
arrangements are necessary to allow for local scoring with 
Testmate? Does the commitment include a timeline? 

H. Written Service Commitment: 

Has the publisher provided written commitment to service? How 
thorough? How responsive? 

I. Flexible Payment Schedule: 

Has the publisher provided written commitment for a flexible 
initial-purchase payment schedule? How flexible? 

A: F/sbO/wp/TESTREV2 . dir i b 
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